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Introduction 


The Information Systems audit report titled CHIMES-EA Combined Healthcare Information and 
Montana Eligibility System — Enterprise Architecture (15DP-01) was issued to the Committee in 
November 2015. The audit included 17 recommendations to the Department of Public Health and Human 
Services (department). In spring 2017, we conducted follow-up work to assess implementation of the 
report recommendations. This memorandum summarizes the results of our follow-up work. 


Background 


CHIMES-EA was implemented in November 2012 as the system to manage the eligibility for 
Supplemental Nutrition Assistance Program (SNAP), Temporary Assistance for Needy Families (TANF), 
Medicaid, and Healthy Montana Kids. CHIMES-EA is used by over 1,000 users including those working 
at Offices of Public Assistance and Processing Centers within the state. The scope of the initial audit 
included SNAP and TANF. Medicaid was being migrated into CHIMES-EA from the old system during 
fieldwork. SNAP and TANF eligibility and benefit amount is automatically determined after the 
applicant’s information is entered into CHIMES-EA. Any reporting or updates to the applicant’s 
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information are also managed through the system. CHIMES-EA interfaces with various other systems for 
applicant information verification and to share information for other state functions, like child support 
enforcement. 


Audit work reviewed system functionality related to SNAP and TANF, performance monitoring, system 
issue management, and user access management. A survey of system users was also conducted. This work 
resulted in 17 recommendations presented in the report. 


Audit Follow-Up Results 


We conducted a follow-up survey to core system users in the Offices of Public Assistance to again gauge 
user perception of the system. Of the 399 users contacted for the survey, 225 users completed the survey 
for a 56 percent response rate. The majority of the core users who responded were client service 
coordinators and supervisors, 207 of the 225. In the initial audit survey, there were 251 responses from 
the same group. 


To obtain an understanding of overall user perception of the system, we asked the following question 
from Table 1 in both the original audit survey and the follow-up survey. When comparing follow-up 
survey responses to the original audit survey, only responses from the core users were included. This 
allowed us to better understand core user perception since the audit. The results to this set of questions 
can be seen in the following table. Based on the survey results, user perception of these system features 
does not appear to have changed since the original audit was issued. 


Table 1 
Audit and Follow-Up Survey Responses: Client Service Coordinators 
and Supervisors 
Audit Follow-up 
eynierera: Always! Most of Always! Most of 
In your opinion, is CHIMES EA... ieume fie uine 
Responding in a timely manner 24% 25% 
User Friendly 23% 23% 
Processing applications quickly 25% 23% 
Increasing the efficiency of collection, 23% 21% 
reporting, and analysis of client data 
Allowing you to spend the majority of your 15% 12% 
time on qualitative customer care 
Easy to learn 34% 30% 
Accurate 31% 27% 
Reducing Errors 18% 12% 
Simplifying Processes 22% 17% 


While survey information was taken into consideration, it was not the only source of information used 
when determining whether each recommendation has been implemented. We also reviewed performance- 
monitoring tools and reports, reviewed system documentation updates, discussed new review and 
monitoring procedures with department staff, and reviewed reports from the issue management system. 
The following sections summarize the progress toward implementation of the report recommendations. 
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RECOMMENDATION #1 


We recommend the Department of Public Health and Human Services improve performance- 
monitoring processes by: 


A. Defining specific goals relative to the system and each program managed by the system. 
B. Defining and documenting a process to consistently review all data relative to these goals. 


C. Reviewing information gathered to identify, document, and track system relation to root 
cause, and communicating results to users. 


Implementation Status — Implemented 


During the audit, there were no metrics or consistent, well-established reviews of TANF equivalent to 
SNAP. The department was in the process of establishing dashboards to review certain metrics 
throughout public assistance. During the audit, multiple dashboards were being developed to track 
federally reported Medicaid and SNAP metrics. Now other metrics are included in the dashboards that 
show an overall picture of SNAP, Medicaid, and TANF, like application processing time, helpline 
waiting times, accuracy, and customer satisfaction. These dashboards are summarized monthly for 
management to review. Through this review, causes of metrics not meeting goals are investigated, 
including system issues. 


To derive the accuracy metric for the dashboard, the department conducts reviews both before and after 
benefits have been authorized or denied from sampled cases every month. TANF and SNAP are included 
in these authorization reviews, but the metrics from this review data will not be broken down to program 
level on the dashboard. The department has further developed a business intelligence tool to provide 
metrics related to all programs, including TANF, such as overpayments, accuracy, and timeliness. 


RECOMMENDATION #2 


We recommend the Department of Public Health and Human Services develop a plan to promptly 
address outstanding system defects that impact eligibility determinations and benefit calculations. 


Implementation Status — Being Implemented 


At the time of the audit, there were around 400 system defects related to eligibility and benefit 
calculations. Twenty-five percent of those, about 100, were related to SNAP or TANF. Audit work also 
showed more enhancements were being addressed than defects with every update to the system. 


Follow-up work identified that the amount of issues affecting eligibility and benefit calculations had 
slightly increased to almost 500 by the end of January 2017. Of these issues, over 200 (46 percent), were 
related to SNAP, TANF, or all programs. While the number of defects has increased, the department is 
addressing them through the issue management and strategic planning processes. This includes dedicated 
staff and bi-weekly system updates. As of February 2017, only 18 SNAP and TANF eligibility and 
benefit defects from recent months do not have business value assessments (BV As). These BVAs help 
determine the priority of when the issue will be addressed. Previous system updates also show more 
defects being addressed than enhancements for the system. It is apparent the department has made 
progress in implementing the recommendation. However, the amount of defects has increased and there 
are still high-priority defects over a year old. User perception of issue management appears to be an 
ongoing issue as well with a third of survey respondents indicating they are rarely or never confident that 
an issue will be taken care of when they report it. While improvements have been made in prioritizing 
these issues, these metrics indicate more progress can be made on fixing outstanding system defects. 
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RECOMMENDATION #3 


We recommend the Department of Public Health and Human Services improve interface 
monitoring by: 


A. Increasing oversight and review of the monitoring process. 


B. Ensuring only necessary information is logged and only users with a business need can 
access logged information. 


Implementation Status — Partially Implemented 


During the audit, personal information was being stored as part of interface logs. These logs contain 
various pieces of data including how many records were matched between two systems, what records 
were not matched, and how many records shared information. Personal information was being logged for 
records that did not match for a certain interface. These logs were stored in the issue management system 
that multiple users had access to. These users had no business need to have access to this information. The 
department has discontinued recording personal information. During follow-up work, we reviewed recent 
interface monitoring documents and identified unnecessary information is not being logged. 


Issues, like unnecessary data being captured, are caught through department oversight and test of control 
measures. With the new interface monitoring tools within the department, automatic queries are ran 
monthly against logs to ensure personal information is not being logged for any interface the tool 
monitors. However, the interface logs identified in the audit are managed by the vendor to identify issues 
so the log is run independently of the monitoring tool and is stored in the issue management system. The 
department does not randomly or periodically check these interface logs to ensure the data being logged is 
correct, or that the vendor is completing a thorough review of interfaces. If this review was done, the 
issue with unnecessary information being recorded would have been identified by the department in a 
more timely manner. 


RECOMMENDATION #4 


We recommend the Department of Public Health and Human Services strengthen interface 
controls by consistently: 


A. Meeting with all stakeholders to document and review current issues. 


B. Communicating with and updating stakeholders throughout the process. 


Implementation Status — Implemented 


Communication with other divisions within the department that shared information for their own business 
processes was minimal during the audit. This led to manual work by other divisions to complete tasks or 
research why the issues were occurring. Since the audit, meetings with other divisions, project 
management, and the vendor have been occurring monthly to address new issues and update current 
issues. Quarterly meetings between divisions occur to address any program or process changes that may 
impact the other division. These meetings get in front of issues that occur because a change was made 
without informing everyone involved. The stakeholders that were interviewed for follow-up work felt the 
meetings have been an improvement and look forward to continuing these collaborative efforts to reduce 
issues. 


RECOMMENDATION #5 


We recommend the Department of Public Health and Human Services conduct a review of 
consolidated, statewide overpayment information, including potential overpayments, on a periodic 
basis. 
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Implementation Status — Not Implemented 


According to department policy, overpayment logs containing potential overpayments that are either 
approved or denied should be consolidated at central office for further review. During the audit, these logs 
were no longer being consolidated because the department was trying to replace them with functionality 
in the business intelligence tool. However, the business intelligence tool was not able to track potential 
overpayments suggested by the system that were denied. These are key to review and understand why an 
overpayment was denied and if it was due to a system error. The previous policy is still in place and the 
logs are still not sent to central office for review. 


The department created an Integrity Unit in summer of 2016 to review overpayments monthly and 
suggest possible improvements. While this is an improvement, it does not meet the overpayment log 
process established in department policy, and user perception of how the system handles overpayments 
appears to still be an issue. Through the Integrity Unit’s own review, it has been identified that receiving 
information kept in these overpayment logs from the Office of Public Assistance (OPA) would be 
beneficial in identifying issues. 


RECOMMENDATION #6 


We recommend that the Department of Public Health and Human Services establish and document 
a process to review metrics related to the source of overpayment claims and address any identified 
system functionality issues. 


Implementation Status — Being Implemented 


At the time of the audit, almost 90 percent of denied overpayments were created by the system while over 
80 percent of approved overpayments were manually created by users. This indicated the system has 
issues with identifying and suggesting accurate overpayments. In the last year, the department has created 
an Integrity Unit that reviews overpayments as one of its main tasks. A report of overpayments that were 
denied, approved, and manually created is reviewed by the unit to identify areas of improvements. These 
are then discussed with management to create actionable items. Survey responses indicate the majority of 
users are not confident changes will occur as result of these reviews. While the integrity unit process is 
working to identify issues, time is still needed to understand how effective the process is, how many of 
the Integrity Unit’s identified issues will be addressed, and how this will impact user perception. 


RECOMMENDATION #7 
We recommend the Department of Public Health and Human Services: 


A. Review claim management processes and controls to continually identify risks and 
opportunities for automated improvements to controls and efficiencies. 


B. Ensure all users are trained appropriately to set up and approve overpayment claims 
according to policy. 


Implementation Status —Being Implemented 


During the audit, the process of managing an overpayment claim after the claim was established was 
somewhat complicated and if changes needed to be made, the process could take 2-3 days before the 
change was complete. These changes were also manually intensive processes. This process is managed by 
the Quality Assurance Division (QAD) and involves both CHIMES-EA and the accounting system within 
the department. QAD has since updated its procedures to simplify the process; however, there has been no 
system changes to improve the process. 
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The department has updated training materials and introduced a new Public Assistance User Guide to 
assist employees on business processes and tasks. However, training at this point is only for new staff and 
the business process documents are still being created. Overpayments do not have a document yet. 
Without clarification on the correct process and the amount of manual overpayments created, the risk for 
incorrect overpayments is increased. This increased risk leads to OPA supervisors spending more time 
than necessary reviewing overpayments and correcting issues. 


RECOMMENDATION #8 


We recommend the Department of Public Health and Human Services ensure all current help desk 
tickets, enhancements, and defects have been prioritized. 


Implementation Status — Implemented 


At the time of the audit, only 25 percent of system defects had been evaluated through the business value 
assessment (BVA) process. This process asks various questions to determine a level of priority for the 
defect. Without this priority determined, defects may not be addressed in an efficient manner. The 
department stated that as of the end of November 2016, there were still 349 BVAs remaining to be 
completed and these would be done by the end of 2016. Follow-up work identified that almost 200 BVAs 
remained to be completed at the end of January 2017. The majority of help desk tickets have also been 
prioritized since the audit, with only 16 tickets prior to this year still needing a priority. While the 
department has made a significant effort to complete these assessments for prior enhancements and 
defects and prioritize help desk tickets, there are still more recent issues remaining that have not been 
evaluated. Most of these BVAs were from the previous 2-3 months. To ensure the current issue 
Management process remains effective, the department should continue progress in completing BVAs as 
defects and enhancements are identified. 


RECOMMENDATION #9 


We recommend the Department of Public Health and Human Services establish, monitor, and 
enforce benchmarks for: 


A. Analysis and prioritization of help desk tickets, defects, and enhancements. 


B. Time frames for implementing solutions. 


Implementation Status —Being Implemented 


During the audit, the department relied on the vendor for major processes throughout the issue 
management cycle. There appeared to be bottlenecks at some of these process points as well. At the time 
of the audit, the average age of defects was 160 days. Since then, the age has increased to over 200 days. 
Survey results also show that 25 percent of users felt issues were always or most of the time handled 
timely, while 27 percent felt issues were rarely or never handled timely. The department has taken on the 
majority of these processes and an “aging” metric has been added when reviewing the BVA scores of 
defects and enhancements to account for timeliness of issues. This metric is used to identify issues that 
are older by assigning a higher value to them. 


While the aging metric increases awareness of issue age, no benchmarks have been set for analysis or 
prioritization of issues and audit work also identified performance standards within the vendor contract 
that are still not being monitored. The department recognizes the performance standards are not the best 
way of monitoring the process and have implemented dashboards monitoring monthly progress in areas 
such as defects assigned to updates, defects that have a BVA, page performance, and other system 
metrics. They intend to incorporate more relevant metrics in the next contract during the rebidding 
process in 2020. 
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While the average age of issues could be reduced by setting such benchmarks, the volatility of the metrics 
at this time makes it difficult to know what the benchmark should be set at. Until then, the department 
needs to continue monitoring issues and the process to improve user perception of issue management 
timeliness. 


RECOMMENDATION #10 
We recommend the Department of Public Health and Human Services: 


A. Improve the process by ensuring all classifications on help desk tickets are accurate and 
complete and defects are associated as necessary. 


B. Document the process to ensure consistency. 


Implementation Status —Partially Implemented 


During the audit, help desk tickets not classified correctly were identified. There are various types of 
issues, like help desk tickets, enhancements, and defects, and each of these has its own classifications. 
These classifications, such as priority, issue resolution, and associated defects, are important in the defect 
prioritization process and help desk ticket review. An example of this is the number of related help desk 
tickets being used as a factor in the assessment to determine overall priority. There was no clear 
documentation of what each classification meant or the process for each type of issue for staff to follow. 
The department has created a document showing the workflow and various stages of each issue; however, 
this document does not provide assistance in determining classifications. 


Defect association to help desk tickets is important in order to understand the impact of the defect. The 
Business Value Assessment (BVA) for defects includes a rating for the number of help desk tickets 
associated to it for this reason. When reviewing recent defects, there were various ratings for this 
question, yet only one help desk ticket associated with the defect. This indicates either a lack of 
knowledge on how to complete this rating or that help desk tickets are not being consistently linked with 
defects. This association provides important information used when completing BV As and is necessary to 
accurately prioritize defects. 


RECOMMENDATION #11 


We recommend the Department of Public Health and Human Services establish a process to limit 
reliance on data fixes and ensure data fixes are associated with a long-term application fix, if 
necessary. 


Implementation Status — Implemented 


The audit indicated there could be an overreliance on fixing the specific data issue, or data defect, instead 
of fixing the system process that created the issue, known as an application defect. At the time of the 
audit, there were over 2,600 closed data defects and only 102 closed application defects, while over 

800 application defects were waiting to be resolved. Currently there are over 8,800 closed data defects 
and 316 closed application defects. Reports also show that currently there are over 900 open application 
defects waiting to be resolved. This is a difference of 6,100 data defects to only 200 application defects 
being resolved in almost 2 years. The department labels defects that consistently occur as recurring data 
defects. This metric is reported on monthly by the vendor. According to the latest vendor progress report, 
there are 28 of these defects with an identified application defect, and 20 that do not have an application 
defect. 


Data defects are necessary in an environment that focuses on timeliness; however, this can easily lead to 
quick fixing of errors without fixing root causes. This creates user frustration when running into the same 
error and decreases efficiency in the issue management process. While the department has a process 
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reviewing recurring data defects, it does not appear to be impacting the number of application defects 
compared to data defects being addressed. The department needs to continue monitoring these metrics to 
ensure the process is effectively identifying recurring data defects and reduces the amount of individual 
data defects being completed. 


RECOMMENDATION #12 


We recommend the Department of Public Health and Human Services increase communication to 
users regarding: 


A. Individual help desk ticket resolution and any related defect resolutions. 


B. Defects being addressed through each build and what workarounds will no longer be 
necessary. 


Implementation Status — Implemented 


The survey conducted during the audit showed that users were frustrated with communication from the 
central office and IT staff about system issues. Audit work identified examples of closed issues that were 
not communicated to users. Users also felt uninformed about system updates and exactly what was 
changing. Since the audit, the department has implemented system pop-ups that alert a user of changes 
whenever they log in. The structure of these pop-ups has gone through a couple changes to ensure any 
language that may not be understood is removed so that it is clear how the change affects the user. 


Follow-up work showed over 40 percent of respondents felt the new system notifications were always or 
most of the time helpful and providing information they need, while around 15 percent of respondents felt 
they rarely or never did so. When asked if they felt communication has increased since the audit, 
respondents who agreed, disagreed, or were undecided were almost evenly split, as shown in Figure 1. 


Figure 1 
Follow-Up Survey Results: Issue Communication 


Has Issue Communication Increased? 


= Agree/Strongly Agree = Undecided = Disagree/Strongly Disagree 


Source: Legislative Audit Division. 


While the department has implemented improvements to issue communication, more time may be needed 
to affect user perception. Over half of respondents feel the new improvements are at most “sometimes” 
helpful and providing information. 
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RECOMMENDATION #13 


We recommend the Department of Public Health and Human Services improve current access 
review procedures by including: 


A. What privileged functions are allowed in each role and if it is necessary. 


B. Which users have roles with access to privileged functions. 


Implementation Status — Implemented 


The user access review performed during the audit showed that while the necessity of a user’s access was 
reviewed every 6 months, what type of access that user had and what functions were allowed to each role 
was not being reviewed. This is important when business processes change to ensure access still aligns 
with the new processes. The department has reviewed privileged functions once since the audit, in 
January 2017, after the follow-up response was received. Privileged functions have been identified and 
noted in the procedure for conducting this review. Staff intend to continue this review every 6 months as 
part of the normal user review. 


RECOMMENDATION #14 
We recommend the Department of Public Health and Human Services: 
A. Document and track contractor access to system databases. 


B. Monitor contractor activity within system databases. 


Implementation Status —Being Implemented 


No current records or documentation of what type of access contractors had to databases was available 
during the audit and contractor activity within state databases was not being monitored for inappropriate 
activity. Since the audit, the department has implemented a system-monitoring tool that automatically 
tracks and alerts activity to the databases and provides reports such as privileged user activity and after- 
hours activity. These reports are sent weekly to security staff; however, at this time, the department 
indicates there is not enough staff to go through the amount of data provided by the new system- 
monitoring tool because it monitors all department systems, not just CHIMES-EA. The department is 
currently working on establishing a process to view all of the data easily and quickly. 


RECOMMENDATION #15 


We recommend the Department of Public Health and Human Services strengthen manual change 
controls by implementing procedures to monitor manual issuances, manual supplemental 
payments, and overrides. 


Implementation Status — Partially Implemented 


Audit work identified that a large number of users had access to high-risk, manual functions. These 
functions are considered privileged functions and if not limited to very specific users, should be 
monitored to ensure they are not misused and are being used accurately. The department now has metrics 
showing the number of manual issuances and overrides as part of the new dashboards; however, there is 
no metric for manual supplements and these reports are not reviewed at lower levels or by user to ensure 
they are not being misused. Access to these functions is limited to supervisors only, which still includes 
almost 80 users. Monthly review of total manual issuances could show statewide trends, but may not 
show if single users are overusing or misusing the privilege. 
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RECOMMENDATION #16 
We recommend the Department of Public Health and Human Services improve user training by: 
A. Covering scenarios that introduce more policy. 


B. Evaluating how satisfied users are with the amount of preparation and training they 
received. 


C. Update training on a regular basis. 


Implementation Status — Being Implemented 


During the audit, users expressed frustration with training and materials provided by the department. 
When we reviewed available materials for guidance when working in the system, the materials lacked 
information in certain areas, including establishing overpayments and self-employment. During the audit, 
the department started requesting user feedback on training and after the audit was presented, the 
department worked with users to identify ways to improve training. This resulted in the department 
restructuring new user training to provide a more in-depth look at the program, policies, and system over 
a 6-week period with 6 weeks of close monitoring following. The first rollout of this training was in 
March 2017. The department stated that ongoing training for more tenured users and any other 
enhancements will be a future project after the new-hire training is complete. The department has begun 
gathering user feedback on what areas to focus continued education on first through regional meetings 
with OPA staff. 


RECOMMENDATION #17 


We recommend the Department of Public Health and Human Services create or update 
documentation and strengthen controls for reviewing documentation in the following areas: 


A. Policy referencing previous systems 

B. Business Rules and Correspondence Detailed Design Documents 
C. Interface Design Documentation and Data Sharing Agreements 
D. Benefit Discrepancy Business Rules and Process 

E. Issue Management Procedures 

F. System Testing Plans 

G. User Guides and System Manuals 


Implementation Status —Partially Implemented 


Throughout the audit, various types of documentation were either out-of-date, incomplete, or did not 
exist. These documents provide baseline configurations, process details and clarification, system 
information and guidelines, as well as verification of process completion. Follow-up work identified that 
policy has been updated and system documentation related to specific defects or enhancements have been 
updated; however, the other documents mentioned in the recommendation still require further updates. 
Specific system documentation was not completely updated and still lacks key components identified in 
the audit. 


User guides and system manuals have been rolled into a single source of information known as the Public 
Assistance User Guide (PAUG). The follow-up survey asked users how effective they feel the new tool is. 
While around 66 percent of survey respondents feel the PAUG answers questions they have about policy, 
procedures, and the system at least sometimes, the other third of respondents feel it rarely to never 
answers those questions. User comments mentioned that the PAUG was too vague, was not updated ina 
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timely manner, was not user friendly, and does not include all processes. The types of needed 
improvements users mentioned included examples of specific policy and more detailed information. 


The PAUG includes business process, policy, and system documentation. Policy is updated; however, not 
all business processes have been documented. Previously, policy included the business process for the 
situation. When policy was updated, these processes were removed to be put into separate documents. 
This could be the source of some of the frustration with users. We realize that user training and 
understanding of how to best utilize the PAUG can also contribute to the survey comments received. The 
department needs to take further steps to inform users, complete the business processes, and address user 
concerns for the PAUG to be successful for more users. 
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Appendix A 


This section provides survey response data. Throughout the survey, users were also given opportunities to 
comment on certain topics. Those comments were not included in this summary. The total population for 


May 2017 


the survey was 399, with 238 responses (225 completed the survey) and 161 nonresponses. Throughout 


the survey, skip logic was used so users were directed to questions relevant to their position and 


interaction with the system. 


Survey Data Used for Follow-Up 


What is your role in the system? 
Total Responses: 
Eligibility Levels 1-5 
Eligibility Supervisor 
Other (please specify in 50 characters) 
CSED Related 
Director 
Investigator/Auditor 
Not sure 


Total Responses: 
Less than 6 months 
6 months - 2 years 
2-5 years 
5-10 years 
More than 10 years 


238 
193 


238 
12 
57 
75 
51 
43 


Program Officer 
How long have you been in your current position? 


Nonresponse: 0 


Nonresponse: 0 


Did you take the survey during the CHIMES audit in Spring 2015? 
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Total Responses: 238 Nonresponse: 0 
Yes 111 
No 67 
| don't remember 60 
In your opinion, is CHIMES... 
Most 
Total Responses: 225, Nonresponse: 13 of the 
Always Time Sometimes Rarely Never NA 

User Friendly 3 51 107 52 12 0 
Responding in a timely manner 2 59 110 50 4 0 
Easy to learn 6 61 77 59 20 0 
Processing applications quickly 3 50 111 46 8 5 
Accurate 0 60 129 30 0 
Simplifying Processes 5 33 102 62 19 2 
Reducing Errors 2 26 90 83 18 4 
Allowing you to spend the majority of your time 
on qualitative customer care 1 28 83 84 22 5 
Increasing the efficiency of collection, 
reporting, and analysis of client data 2 46 96 61 13 3 
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How often does CHIMES support you when completing your daily tasks? 


Total Responses: 225 Nonresponse: 13 
Always 19 
Most of the Time 82 
Sometimes 91 
Rarely 33 
Never 0 


The division has recently established dashboards and reports to monitor key metrics related to being 
respectful, prompt, and accurate. Specific to SNAP and TANF metrics and goals, do the new reports 
and dashboards: 


Most 
Total Responses: 221, Nonresponse: 17 of the 
Always Time Sometimes Rarely Never NA 
Provide users with important information 11 54 68 40 12 36 
Provide insight to management's decisions 4 23 59 51 34 50 
Provide helpful information that | relate to and 
can understand 4 44 63 53 20 37 


What best describes your role in CHIMES? If more than one apply, select the statement that represents 
most of your interaction with CHIMES. 


Total Responses: 220 Nonresponse: 18 
| enter information in to the system 183 
| supervise/manage people who enter 
information in to the system 20 
| view information, but do not enter information 4 
Other (please specify in 50 characters) 13 
The system: 
Most 
Total Responses: 202, Nonresponse: 36 of the 
Always Time Sometimes Rarely Never NA 

Identifies potential overpayments 2 46 100 45 5 4 
Calculates the correct overpayment amount 3 47 78 58 11 5 
Reduces the amount of time spent 
establishing an overpayment 2 29 63 65 38 5 


Recently the department has established the Integrity Unit to review cases, including overpayments. 
This new review has: (select all that apply) 


Total Responses: 202 Nonresponse: 36 

Improved CHIMES’ ability to identify and 

calculate overpayments and underpayments 9 

Improved user understanding of overpayment 

issues 10 

Improved the overpayment process 18 

| have not seen any changes as a result of the 

Integrity Unit review 177 

If you have not seen any changes, how confident are you that improvements will occur in the future? 
Total Responses: 175 Nonresponse: 63 

Extremely confident 5 

Somewhat confident 62 

Little confidence 80 

No confidence 28 
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How often do you manually perform the following actions because the system did not act as intended? 


Most 
Total Responses: 200, Nonresponse: 38 of the 
Always Time Sometimes Rarely Never NA 
Open/Close a case 7 33 122 25 8 5 
Withdraw an application (due to the system 
not acting as it should) 2 21 107 52 14 
Override benefit amount 2 14 125 44 8 
Issue benefits 8 17 124 39 
Other manual operations due to the system 
not acting as it should. (please specify in 80 
characters) 39 
Would you agree that your use of these manual changes is decreasing? 
Total Responses: 200 Nonresponse: 38 
Strongly Agree 4 
Agree 44 
Undecided 73 
Disagree 55 
Strongly Disagree 19 
| don't make manual changes s) 
Are you required to report issues found in CHIMES? 
Total Responses: 216 Nonresponse: 22 
Yes 180 
No 36 
Whenever there is an issue in CHIMES: 
Most 
Total Responses: 179, Nonresponse: 59 of the 


Always Time Sometimes Rarely Never NA 
| report the issues even if | know a way 


around the issue 27 57 63 27 5 0) 
It is being addressed in a timely manner 12 32 83 45 4 ) 
| feel confident it will be taken care of 20 31 68 46 12 0 


The department has implemented release notes and system pop-ups to inform users of changes and 
issue resolutions for CHIMES. How often are these new notifications: 


Most 
Total Responses: 179, Nonresponse: 59 of the 
Always Time Sometimes Rarely Never NA 

Helpful 17 59 78 23 2 0 
Providing the information | need about the 
change 15 62 70 28 4 0 
Increasing communication about system 
issues 21 46 72 32 8 0 


Please select what communication you have received concerning issues you have reported. (select all 
that apply) 


Total Responses: 179 Nonresponse: 59 

Issue or Ticket number 176 
Request/Follow-up for Issue details 82 
Update on issue status 63 
Notification of when the issue will be fixed 63 
Reason for closing the issue (fixed or not) 118 
| don't report issues 1 

Other (please specify in 50 characters) 15 
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Do you agree that communication about issues you report is increasing? 


Total Responses: 179 Nonresponse: 59 
Strongly agree 6 
Agree 50 
Undecided 73 
Disagree 37 
Strongly Disagree 13 


The department has created the Public Assistance User Guide to assist users. How often does this 
answer questions you have about: 


Most 
Total Responses: 179, Nonresponse: 59 of the 
Always Time Sometimes Rarely Never NA 
How CHIMES enforces policy 3 36 80 46 14 0 
How you should use CHIMES 5 32 81 47 14 0) 
Department procedures involving CHIMES 5 31 81 49 13 0 
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